This is an automated cherry-pick of #10637

What problem does this PR solve?

Issue Number: close #10633, ref #10557

Problem Summary:
Currently, TiFlash Compute’s FileCache downloads data from S3 into local block storage as follows: it obtains an std::istream object via S3’s GetObject API and requests permission from the RateLimiter based on the total size of the entire S3 object before writing (implemented by #10555). Once the rate limiter approves the request, the full object is written to the local block storage in one burst.
Although this approach may keep the average write throughput below the quota threshold—making it appear compliant—the instantaneous write rate during the actual download can temporarily exceed the quota. This burstiness can trigger throttling by the underlying block storage, leading to elevated I/O latency.

What is changed and how it works?

disagg: Fine-grained rate limiting for S3 cache downloads
  * Use 128KiB buffer for S3 cache downloads and rate limiting. Avoid burst IO that cause block storage throttling

Cloud vendors usually consider 16KiB as 1 IOPS

• AWS: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html
• Alibaba Cloud: https://www.alibabacloud.com/help/en/ecs/user-guide/essd-autopl-disks

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

1. deploy a next-gen cluster
2. load chbenchmark-1500 dataset and add tiflash replica
go-tpc tpcc prepare --warehouses 1500 -T 30 -D chbenchmark -H 172.31.7.2 -P 4000
go-tpc ch prepare -H 172.31.7.2 -P 4000 -D chbenchmark1k5_clustered 
# deploy 2 tiflash-compute with limited resources and ebs bandwidth
# m7a.2xlarge (8c32G), IOPS 6000 Throughput 125
# patch 1 tiflash-compute with the master
tiup cluster patch ng ./tiflash-patch.965cc8.tar.gz -N 172.31.10.1:9000 -y
# patch 1 tiflash-compute with this branch
tiup cluster patch ng ./tiflash-patch.8096c4.tar.gz -N 172.31.10.2:9000 -y
# shutdown 172.31.10.2 and run chbenchmark on master branch tiflash-compte
go-tpc ch run -H 172.31.7.2 -P4000 --warehouses 1500 -D chbenchmark1k5_clustered -T 50 -t 2 --time 15m
# shutdown 172.31.10.1 and start 172.31.10.2 to run chbenchmark on this branch
go-tpc ch run -H 172.31.7.2 -P4000 --warehouses 1500 -D chbenchmark1k5_clustered -T 50 -t 2 --time 15m
# check the metrics

Performance changes

the read latency is also slightly reduced

# master branch
tpmC: 4218.7, tpmTotal: 9365.9, efficiency: 21.9%
[Summary] Q1     - Count: 133, Sum(ms): 4574862.2, Avg(ms): 34396.9
# this PR
tpmC: 4486.4, tpmTotal: 9974.4, efficiency: 23.3%
[Summary] Q1     - Count: 133, Sum(ms): 4547837.2, Avg(ms): 34194.5

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes

  • Safer S3 download flow using temp-file-then-rename to avoid partial/corrupt files and preserve capacity accounting.

• Improvements

  • Consolidated download streaming into a single write-limited local path, improved timing precision, INFO-level logs, and updated download metrics.
  • Added debug-mode size sanity checks and clearer segment state transitions on failure/success.

• Tests

  • Expanded eviction and background-download tests with additional scenarios, assertions, and instrumentation.

• Public API

  • Renamed segment completion setter and added a status getter for clearer segment state handling.

• Chores

  • Updated .gitignore to ignore a new secret file.

_{✏️ Tip: You can customize this high-level summary in your review settings.}